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ABSTRACT 


Ship performance characteristics, such as max-sustained speed, acceleration, and 
maneuverability are generally pre-determined as a platform requirement based on 
precedents. However, these pre-determined performance characteristics have far 
reaching impacts on the size, logistics, manning, and cost of the ship platform. Instead of 
designing to pre-defined platform performance requirements, ship performance 
characteristics should be determined based on fulfilling mission objectives 

This research evaluates the viability to effectively determine if the ship 
characteristic requirements can be quantified by using Naval Sea Systems Command’s 
Naval Battle Engagement Model (NABEM)—an agent-based simulation tool developed 
by Naval Sea Systems Command. In particular, we study two tactical situations by 
varying three platform characteristics—maximum speed, acceleration, and turning 
diameter—and determine how these platform characteristics affect mission performance. 
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EXECUTIVE SUMMARY 


Ship performance characteristics, such as max-sustained speed, acceleration, and 
maneuverability are generally pre-determined as platform requirements based on 
precedents. However, these pre-determined performance characteristics have far 
reaching impacts on the size, logistics, manning, and cost of the ship platform. Instead of 
designing to pre-defined platform performance constraints, ship performance 
characteristics should be designated to fulfill mission objectives. 

This research evaluates the viability to effectively determine if the ship 
characteristic requirements can be quantified by using Naval Sea Systems Command’s 
Naval Battle Engagement Model (NABEM)—an agent-based simulation tool developed 
by Naval Sea Systems Command. In particular, we study two tactical situations by 
varying three platform characteristics—maximum speed, acceleration, and turning 
diameter—and determine how these platform characteristics affect mission performance. 

The response surfaces generated from the NABEM simulations produced 
insignificant results. Maximum speed, acceleration and turning diameter produced only 
secondary effects. NABEM is primarily an engineering simulation model, concerned 
more with the detailed mathematical representation of individual systems or components. 
It provides a detailed representation of sensor and weapon systems, not platform 
characteristics. For this reason, it appeared that sensor and weapons have a stronger 
effect on the results, while maximum speed, acceleration and turning diameter appeared 
as secondary effects producing little or no effect. 

Implementation of the methodology allows a designer to assess and trade-off 
impacts of various ship characteristics based on mission effectiveness. The methodology 
provides a framework where feasible and economically viable alternatives can be 
identified with accuracy along with their effects on mission effectiveness. While the 
methodology is capable of supporting the JCIDS process, NABEM was not an effective 
simulation tool. 

Maximum speed, acceleration and turning diameter are typically tactical decisions 
made by the ship operator based on the current tactical situation, the “human in the loop”. 
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If the platform sensors and weapons generate the tactical decision environment, then ship 
platform performance—such as maximum speed, acceleration, turning diameter—may 
become the dominating factors, which makes it possible to quantitatively assess their 
effects on mission effectiveness. Modeling human response to changing tactical 
situations, in relation to platform performance, is a daunting task. Such a model would 
greatly enhance future ability to assess ship platform characteristics on mission 
effectiveness and is suggested as a future research project. 



I. 


INTRODUCTION 


Ship characteristics are generally determined based on historical precedent. 
Characteristics such as max sustained 1 speed, signatures 2 , acceleration, and maneuvering 
are picked based on what was done previously, unless that characteristic was proven bad 
or needed improvement. However, such decisions can have far reaching impacts to the 
platform size, lifetime logistics, manning, and cost. There must be a better way to 
quantitatively determine ship characteristics that directly relate the ships ability to 
perform its assigned mission. The objective of this work is to determine a framework 
that can quantify these ship characteristics as they relate to operational effectiveness. 

Operational effectiveness relates operational capability to operational 
performance in the form of Measures of Performance metrics (MoPs). However, these 
MoPs must be evaluated within a specific mission and operational environment to be 
meaningful. The purpose of this analysis is to do just that, within the proposed 
framework, determine the effect specific ship characteristics have on the overall 
operational effectiveness during a particular mission. 

A. BACKGROUND 

The U.S. Navy has shifted its emphasis from design to developing broad ship and 
fleet architectures in order to develop design requirements to meet future fleet 
architectures as well as deciding on the merit of future technologies to be pursued. The 
establishment of the Joint Capabilities Integration and Development System (JCIDS) by 
the Chairman of the Joint Chiefs of Staff [13] supports this shift in emphasis. The JCIDS 
process requires a more system approach to determining new system development. The 
process is not centered on the platform or component level system, but on the integration 
and impact of these systems on joint/global force operations, doctrine, organization, 
training, personnel, material, and facilities. Assessing the impacts and effectiveness of 
these complex systems becomes an increasingly challenging problem. System demands, 

1 Max Sustain speed is defined as the maximum speed that can be obtained at 80% of the ships full 
power. 

2 Ship signatures include radar cross section, infrared signatures, and acoustic signature. 
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including increased performance, lower system life cycle costs, longer operating 
capacities, and improved productivity and efficiency, must be balanced against limited 
resources, scant or unknown data, the identification and resolution of conflicts, and 
resource allocation (people and cost). 

These tradeoffs point to the need for an integrated and systematic framework that 
can assess system characteristics as it affects overall system effectiveness. The goal of 
this research was to develop an analysis framework that supports the JCIDS process by 
providing a methodology that provides a trade off environment that relates platform 
performance characteristics to operational effectives in a combat environment. This 
allows a sponsor to justify (quantitatively) operational, material, and technology 
requirements in a better/clearer framework in which to develop the systems level 
requirements and identify future technology investments. 

B. DISCUSSION 

In order to provide the proper framework in which to develop and evaluate 
platform performance characteristics and MoPs, a set of operational and mission 
requirements are generated. These requirements are in the form of Tactical Situations, 
which provide the operational environment, mission characteristics, goals, tasks, and 
threats. These tactical situations provide the framework to develop simulations to 
determine the MoPs, therefore relating operational effectiveness to system performance. 

To ensure that the full range of possible platform characteristics are addressed, an 
operational effectiveness trade space is developed using a Design of Experiments (DoE) 
methodology coupled with a Response Surface Model (RSM). 

C. OBJECTIVES 

The objective of the research is developing a trade space model that relates 
operational effectiveness to platform performance characteristics. In addition, provide 
traceable linkages between measures of performance associated with individual platform 
characteristics, and measures of effectiveness associated with the required mission. 
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II. 


METHODOLOGY 


The operational effectiveness trade space was developed in five steps: (1) 
defining the problem, (2) determining operational measures of performance and metrics, 
(3) modeling and simulation using NAVSEA’s Naval Battle Engagement Model, (4) 
design of experiments, and (5) generating a response surface model. This approach is 
similar to that developed by the Aerospace Systems design Laboratory at Georgia 
Institute of Technology and is know as the Unified Tradeoff Environment 
[8][9][10][11][12][17] and assessed the impacts of system requirements on the system 
design trade space. 

A. PROBLEM DEFINITION 

In order to formulate the problem, it is assumed that there was a need for a new 
class of surface combatant. The Initial Capabilities Document outlines the general ship 
capabilities, but left several ambiguous requirements. These subjective and sometimes 
“fuzzy” requirements must, or should be, mapped into definitive requirements. The 
problem therefore is to develop an analysis framework to best determine these discrete 
design requirements. For this analysis three basic ship characteristics are evaluated, 
maximum speed, acceleration, and turning diameter. These characteristics generally have 
significant cost and ship systems impacts. Determining their impact on mission 
effectiveness would help define their relative importance to the overall ship system. 

B. MEASURES OF PERFORMANCE 

In order to make logical decisions and choices for the three ship characteristics, 
criteria to measure the value or relative importance of alternative characteristics are 
needed. This is an essential part of an operational effectiveness trade space, knowing 
what metrics are to be used to determine operational success or failure as well as how to 
quantify these metrics. 

For this thesis the Measures of Performance (MoP) correspond to individual 
mission performances. Typically MoPs are quantitative and consist of a range of values 
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about a desired point. These values are performance metrics that the mission targets, by 
changing system characteristics, so as to finally achieve the qualities desired for the 
overall mission. MoPs are related to specific missions (i.e., tactical situations) and 
mission tasks. 

The set of MoPs developed for each tactical situation is derived form the 
Universal Joint Task List (UJTL) and the Navy Tactical Task List (NTTL) [6] [7]. The 
UJTL and the NTTL provides relationships between missions, operations, and tasks. 
These relationships, along with the operational analysis, identify the operations and tasks 
that must be performed for mission success. 

The mission establishes the requirement to perform tasks and provides the context 
for each task performance (including the conditions under which a task would be 
performed). It determines where and when a task must be performed (one or more 
locations). Finally, it determines the degree to which a task must be performed (implied 
in the concept of the operations) and provides a way to understand precisely how the 
performance of a task contributes to operational success. 

This thesis studies two tactical situations: Tactical Situation 1, defense of a major 
seaport, and Tactical Situation 2, defense of a coastal convoy. Each tactical situation is 
designed to stress the ship characteristics under consideration. Using the UJTL and 
NTTL, the four general MoPs were determined to be appropriate for the analysis, Table 
1 . 



Units 

Measure of Performance 

M0.1 

Number 

Of Blue ships damaged by enemy attacks 

M0.2 

Number 

Of Blue ship sunk by enemy attacks 

MO. 3 

Number 

Of attacking Red ships damaged. 

M0.4 

Number 

Of attacking Red ships destroyed. 


Table 1. Notional Measures of Performance 
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C. MODELING AND SIMULATIONS 

NAVSEA’s Naval Battle Engagement Model (NABEM) is used to assess the ship 
characteristics impacts on mission performance,. NABEM is the primary tool for the 
quantitative portion of this analysis. NABEM is a sophisticated Monte Carlo, time-step, 
many-on-many warfare model capable of accurately simulating all tactical interactions 
from the sea surface to the upper limit of the atmosphere (air-to-air, air-to-surface, 
surface-to-air, and surface-to-surface engagements). The model can handle any 
combination of air, surface, and shore-based platforms and associated weapons, and also 
includes neutral surface and air traffic (merchantmen, fishing boats, airliners, etc.) when 
desired. 

Created in the 1970s, NABEM has been in continual use at Naval Surface Weapons 
Center, Carderock Division (NSWC-CD) since the 1980s. It has undergone continual 
development and upgrades to maintain its viability and extend its capabilities. NABEM 
is sensitive to a ship’s radar cross section and infrared, visual, and emissions signatures in 
all phases of an engagement for both aircraft and anti-ship cruise missiles including 
detection, targeting, and lock-on. NABEM is also sensitive to ship passive protection 
(vulnerability) and other hull, mechanical, and electrical (HM&E) technology related 
issues. 

The user can script the initial behavior of ships and aircraft, governing their 
movement, rules of engagement, EMCON status, etc. Movement can be randomized to 
any extent desired, allowing NABEM to vary the scenario geometry. NABEM has a 
limited Artificial Intelligence capability, in which both blue and red units act only on the 
information they possess: that is, platforms in NABEM never operate with a “gods-eye” 
view, as often occurs in other models. Platforms in NABEM can operate only on their 
situational awareness at the platform and force level, so that tactical and targeting 
decisions are based on the information held by each platform at each particular moment. 

The model allows for the representation of the events occurring in an engagement 
in a Naval Tactical Data System (NTDS)-like graphics display. This capability is a 
valuable tool for evaluating and validating initial scenario geometry and tactics, tracing 
key events, troubleshooting, and demonstrating the model in briefings. 
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D. DESIGN OF EXPERIMENTS 


The Design of Experiments (DoE) is a statistical driven process that allows for the 
maximization of experimental data. With a minimum number of trials (in this case 
simulation runs), a large number of system parameters can be quantitatively examined to 
understand the effect each parameter has on the overall system. 

The analysis concerns the effect maximum speed, acceleration, and turning 
diameters have on mission effectiveness. These three parameters are the three 
independent input variables used for the DoE. 

1. Maximum Speed 

Maximum speed is varied as an independent variable; measured in knots. This 
variable will determine if ship speed contributes to the overall outcome of the tactical 
situation. The ability of the ship to provide power to maintain max speed will ultimately 
be compared to the outcomes of the tactical simulation to produce a response surface. 

The ranges of speeds for the DoE are listed in Table 2. Max speeds were chosen 
to bracket current and potential surface combatant max speeds. 



Low 

High 

Max Speed (kts) 

25.0 

45.0 

Acceleration (kts/sec) 

0.1 

0.5 

Turning Diameter (ft) 

1000 

2500 


Table 2. Speed Range of Variations 


2. Acceleration 

A range of acceleration is varied as independent variable; measured in knots per 
second. This variable determines if the ships acceleration contributes to the overall 
outcome of the tactical situation. The ability of the ship to provide power to accelerate 
will ultimately be compared to the outcomes of the tactical simulation to produce a 
response surface. 
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The ranges of accelerations for the DoE are listed in Table 2. Accelerations are 
chosen to bracket current and potential surface combatant accelerations. 

3. Maneuvering 

Turning diameter varies the ships maneuvering characteristics, measured in feet. 
This variable will determine if maneuvering affects the overall outcome of the tactical 
situation. During mission execution, the ship must make maneuvers to avoid or engage 
threats. 

The ranges of turning diameters are listed in the Table 2. Turning diameters high 
and low values are based on current and potential surface combatant turning diameters. 

The independent variable ranges for maximum speed, acceleration and tuning 
diameter represent technically feasible solutions for the ship platforms being considered. 
When the region of interest is the same as the region of feasibility, the best DoE model to 
chose is a design cube model. 

There are several different design cube models to choose from, each having their 
own pros and cons. For this analysis a face-center central composite design was chosen. 
The face-center central composite design is an efficient design that is ideal for sequential 
experimentation and allows a reasonable amount of information for testing of model fit 
while not involving an unusually large number of design points. 

Each DoE, one for Tactical Situation 1 and one for Tactical Situation 2 
systematically varies these three independent variables. Table 3. 
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Design 

Points 

Max Speed 
(kt) 

Acceleration 

(kt/sec) 

Turn Dia 
(ft) 

1 

25 

0.5 

1000 

2 

45 

0.5 

1000 

3 

45 

0.3 

1750 

4 

45 

0.1 

1000 

5 

25 

0.5 

2500 

6 

45 

0.1 

2500 

7 

35 

0.1 

1750 

8 

25 

0.3 

1750 

9 

25 

0.1 

1000 

10 

35 

0.3 

1750 

11 

35 

0.5 

1750 

12 

45 

0.5 

2500 

13 

35 

0.3 

1000 

14 

35 

0.3 

2500 

15 

25 

0.1 

2500 


Table 3. Design of Experiments Independent Variables 


Each design point is simulated in NABEM for 5000 independent runs. The 
resulting statistics from each design point provides the operational/mission effects or 
MoPs related to the variations in ship characteristics 

E. RESPONSE SURFACE MODEL 

The RSM is a multi-variable regression technique that models the response of a 
complex system using a simple equation. The response surface is modeled using a 
second-order quadratic equation thus giving a model of the relationships between the 
independent (input) variables and the responses obtained for the simulation model, in this 
case NABEM. The response surface is modeled using a second-order quadratic equation 
and is expressed as 


*=*.+i».*/+£v?+z £v*o 

i =1 i =1 i =1 7=1+1 

Where: by are the coefficients for the cross-product terms. 

bi are the regression coefficients for the first-degree x* and Xj are the design variables. 

terms. bn are the coefficients for the pure quadratic terms. 
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The RSM is generated using data provided by the DoE and the NAMBEM 
simulations. This data is fed into SAS’s JMP software to generate the RSM. JMP 
(Version 5.1) is a statistical analysis software tool that links statistics with graphics to 
interactively explore, understand, and visualize data. The software is designed to uncover 
relationships and outliers in the data. JMP provides statistical tools as well as Design of 
Experiments and Statistical Quality Control. JMP’s built-in capabilities are used to 
develop the DoE, the RSMs, and JMP’s interactive graphic tools to explore the design 
space. In particular JMP’s profiler, contour profiler, and surface graphic displays are 
used. 

F. ANALYSIS OF RESPONSE SURFACE MODEL 

The response surfaces, of RSM, form the basis or framework of the operational 
effectiveness trade space model. Through visualization tools built into the analysis tool 
(JMP), the trade space can be analyzed. The results are displayed through a series of 
visualization tools, prediction and contour profilers, and response surface plots. The 
profiler plots displays prediction traces (predicted responses as one variable is changed 
while holding the others constant) for each variable. The response surface plots provide a 
quick visual of how the response functions are behaving. 

The prediction profiler isolates the impact of every factor for every response. The 
contour profiler illustrates interaction of one or several of the variables have on one 
another. The prediction and contour profilers are interactive plots, therefore, very difficult 
to show in a written report. Therefore, only the general ship characteristics effects on 
mission performance are described for each tactical situation. 

G. ASSUMPTIONS AND CAVEATS 

A conceptual surface combatant is developed specifically for this study and does 
not relate to any program of record or any potential U.S. Navy program. This conceptual 
ship is designated as the Small Surface Combatant (SSC) for this analysis. Mission 
systems for the SSC were modeled to be consistent with current or projected systems and 
are constant and consistent throughout the different tactical situations. To maintain the 
unclassified natural of this thesis, sensor and weapon performance data is generated from 
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open source literature. It should be noted that the intent of the analysis is not to evaluate 
mission systems performance or to compare individual mission system capabilities, 
performance, or effectiveness. The focus of the analysis is to determine the effect of 
specific ship characteristics on the ship’s overall mission performance. 

The Red Force combat systems and platforms are chosen to be representative of 
likely current and future threats; again data is obtained from open source resources. 

The tactical situations reflect missions that stress the ship characteristics being 
evaluated. They are designed to resemble possible real world situations, but the locations 
of these scenarios are kept generic to maintain the unclassified nature of this analysis. 

NABEM is a good combat simulation model; however, like most simulation 
models it has its limitations. One of these limitations is that the simulation has to be well 
scripted, not allowing for a human in the loop decisions as to course, speed, weapons to 
use, etc. that can determine the outcome of a simulation. 

Blue force platforms does not include helicopter capability and is not included in 
the NABEM simulation. 
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III. 


TACTICAL SITUATION 1 


Tactical Situation 1 (TS1) models the defense of a major seaport against enemy 
forces trying to gain entry into the port. Blue force’s mission is to prevent enemy forces 
(Red Force) from gaining access to the port. The port of San Francisco is chosen for this 
tactical situation. 

Blue forces are deployed at the mouth and within San Francisco harbor in 
preparation of Red hostilities. Three blue small surface combatants are assigned a patrol 
area each. Two patrol the approaches to the harbor, while the third patrols the inner 
harbor area, as shown in Figure 1. Figure 2 shows the TS1 scenario as modeled in 
NABEM. 



Blue Force Defensive Patrol Zones 


Red Attack - South 


Red Attack - North 


Red Attack - West 


Figure 1. TS1 General Laydown 
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Red Force-West 


Red Force-South ^ 

Figure 2. TS1 NAB EM Model Laydown 


Blue Force-Patrol Areas 


Red Force-North 

X 


The red force comprises three separate squadrons, each squadron approaching the 
harbor from different directions, as shown in Figure 1. The north and the south 
squadrons had three high speed attack craft, while the western squadron had six. The 
mission goal for Red Forces goal is to penetrate the harbor defenses and disrupt harbor 
operations. 

A. RED FORCE 

The red force is chosen to be representative of the likely threat for this tactical 
situation. Each red force platform, designated as Red PTG, was modeled as a small fast 
attack craft with operating characteristics similar to the Peoples Republic of China Type 
083 Fast Attack craft and the Iranian Navy’s Bohammer fast attack craft. Red PTG has a 
maximum speed of 35 knots, 20.0 nautical mile range navigational radar, and an ESM 
detection system. 

Each Red PTG carries six short-range missiles. Each missile has a maximum 
operating range of approximately 2.0 nautical miles and has similar operating 
characteristics to hand-launched anti-armor missiles and rocket propelled grenades. 
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B. BLUE FORCE 


Blue forces consist of three small surface combatants, designated as Blue SSC. 
Each Blue SSC platform is modeled as a corvette size combatant similar to the German 
Type 143 and 148 Fast Attack Craft and the Swedish Goteborg Class Corvettes. Blue 
SSC forms the basis for the design of experiments in which max speed, accelerations and 
turning diameter are varied. 

Each Blue SSC has a medium caliber gun (similar to the Mk 57 Naval Gun), 24 
short range missiles, a point defense gun (similar to the Mk 15 CIWS), an ESM detection 
and decoy system, and a 200 nautical mile surface search radar (similar to a SPY-IF 
radar). 


C. METRICS FOR TACTICAL SITUATION 1 

The MoPs chosen for TS1 reflect measurable mission characteristics that are 
influenced by the three ship characteristics being investigated (Table 4). The metrics are 
designed to evaluate the effectiveness of the Blue Forces against the threat in each 
particular scenario. 



Units 

Measure of Effectiveness 

Ml.l 

Number 

Red PTGs Survive and Enter Harbor 

Ml.2 

Number 

Blue SSCs Sunk 

Ml.3 

Number 

Blue SSCs Damaged 

Table 4. M 

easures of Effectiveness for TS1 


D. ANALYSIS FOR TACTICAL SITUATION 1 

The DoE is run in NABEM according to the DoE matrix in Table 3. There are 15 
design points; each design point is simulated 5000 times, each with the initial simulation 
parameters randomly chosen. The simulations results are listed in Table 5. Response 
surface models for each MoE are generated from the results listed in Table 5. Each 
response is checked for RSquare values, Adjusted RSquared values, F statistics, and 
Model Fit Error to ensure the accuracy of the model fit. If any of the checks fails, it is an 
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indication that the basic second-order model is not appropriate for the response. In these 
cases two options are available to provide a better fit, adding higher terms or adding more 
design points. For all the MoEs only one of the checks fail, Model Fit Error. Higher 
order terms are tried, but fail more than one of the checks. Generating more design 
points was considered, but due to time and resources was not possible. For this thesis 
the response surface models generated from the design points in Table 6 are the best fit 
possible. For a more thorough examination the TSl’s response model are referred to 
Appendix A. 

E. RESULTS 


Design 

Points 

Max Spd 

Accel 

Turn Dia 

PTGs Enter 

Harbor 

SSCsSunk 

SSCs Damaged 

(kt) 

(kt/sec) 

(ft) 

Mean 

% of Total 

Mean| 

% of Total 

Mean 

% of Total 

1 

25 

mgm 


0.979 

(8.2) 

0.330 

(11.0) 

1.141 

(38.0) 

2 

45 

mm 


1.149 

(9.6) 

0.413 

(13.8) 

1.410 

(47.0) 

3 

45 

0.3 

1750 

1.224 

(10.2) 

0.392 

(13.1) 

1.404 

(46.8) 

4 

45 

0.1 

1000 

1.202 

-10 

0.404 

(13.5) 

1.421 

(47.4) 

5 

25 

0.5 

2500 

1.021 

(8.5) 

0.317 

(10.6) 

1.124 

(37.5) 

6 

45 

0.1 

2500 

1.161 

(9.7) 

0.382 

(12.7) 

1.424 

(47.5) 

7 

35 

0.1 

1750 

0.883 

(7.4) 

0.377 

(12.6) 

1.299 

(43.3) 

8 

25 

0.3 

1750 

0.998 

(8.3) 

0.320 

(10.7) 

1.157 

(38.6) 

9 

25 

0.1 

1000 

0.857 

(7.1) 

0.313 

(10.4) 

1.170 

(39.0) 

10 

35 

0.3 

1750 

0.926 

(7.7) 

0.365 

(12.2) 

1.304 

(43.5) 

11 

35 

0.5 

1750 

0.938 

(7.8) 

0.334 

(11-1) 

1.137 

(37.9) 

12 

45 

0.5 

2500 

1.217 

(10.1) 

0.393 

(13.1) 

1.401 

(46.7) 

13 

35 

0.3 

1000 

0.924 

(7.7) 

0.377 

(12.6) 

1.281 

(42.7) 

14 

35 

0.3 

2500 

0.936 

(7.8) 

0.370 

(12.3) 

1.305 

(43.5) 

15 

25 

0.1 

2500 

0.926 

(7.7) 

0.319 

(10.6) 

1.113 

(37.1) 


Table 5. Final Results for TS1 


From the response surface model for each MoE a prediction profiler plot is 
generated which shows the interrelationships between all the parameters, as shown in 
Figure 3. The prediction profiler serves as the principal tool for evaluating the individual 
responses. The following sections discuss the key findings for each of the ship 
characteristics investigated. 
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Figure 3. Prediction Profiler Results for TS1 


1. Max Speed 

Referring to the prediction profiler in Figure 3; as maximum speed increases all 
three MoEs increase. Meaning, as Blue SSC’s speed increases the number of Red PTGs 
successfully entering the harbor also increases, and so does the number of SSCs damaged 
or sunk. This behavior seems to contradict general conventional rules of thumb, but as 
speed increases. Blue SSCs are exposed to more Red PTGs. So why then does the 
number of PTGs surviving go up and not down with similar weapons and sensors. 
Simple Red PTGs outnumber the Blue SSCs by 4 to 1. Therefore, there is a higher 
probability that Blue will incur damage or sink. With fewer Blue ships available the 
probability that more PTGs survive to complete their mission (enter the harbor) increases. 
Another factor to consider, as modeled in the simulation, Blue SSCs have a limited 
amount of ready service ammunition. As Red PTGs attack, Blue deplete their ready 
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service ammunition. The simulation model reloads ammunition over time and this time 
delay is a contributing factor to the increased number of Blue SSCs damaged or sunk. 

While maximum speed has an effect on the outcome of the tactical situation, the 
effect is insignificant. 

2. Acceleration 

Referring to the prediction profiler in Figure 3, as acceleration increases, the 
number of Blue SSCs sunk or damaged (Ml.2 and Ml.3) tend to decrease and the 
numbers of Red PTGs survive to enter the harbor (Ml.l) increase. This behavior seems 
to contradict general conventional rules of thumb. However, tactical situation 1 iss set in 
a very confined operating space (San Francisco Harbor) negating possible advantages 
acceleration might provide in an open ocean operational area. In addition, the increase or 
decrease in acceleration effects on the MoE are very small compared to the other ship 
characteristics. Combined the overall effect from acceleration is negligible. 

3. Turn Diameter 

Referring to the prediction profiler in Figure 3, as the turning diameter increases, 
the number of Blue SSCs sunk or damaged (Ml.2 and Ml.3) tend to decrease and the 
number of Red PTGs survives to enter the harbor (Ml.l) increase. This behavior also 
seems to contradict general conventional rules of thumb. However, the initial turning 
diameter of 1000 ft produces a high number of Blue SSCs damaged or sunk. This is 
consistent with the results when maximum speed is increased. A highly maneuverable 
ship in confined waters increases the probability of exposure to enemy weapons, 
therefore increasing the probability of damage or sinking Blue SSCs. Factor in the Red 
PTGs four to one superiority in numbers the results can be understood. These effects are 
very small and are insignificant. 

F. CONCLUSIONS 

The final response surface models developed for tactical situation 1 produced 
tangible models that link ship characteristic to mission effectiveness; however, the effects 
are small and insignificant. NABEM is primarily an engineering simulation model, 
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concerned more with the detailed mathematical representation of individual systems or 
components. NABEM provides a detailed representation of sensor and weapon systems, 
not platform characteristics. For this reason sensor and weapons effects dominate the 
results. Maximum speed, acceleration and turning diameter appear as secondary effects 
producing little or no impact to the overall outcome of the tactical situation. 
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IV 


TACTICAL SITUATION 2 


Tactical Situation 2 (TS2) models the defense of a blue convoy transiting through 
a wide strait off the coast of Blue territory. Red forces attack the convoy from two 
different directions and at different times. For this tactical scenario the Straits of Florida 
is modeled, with the Blue convoy, comprised of ten merchant ships, transiting north 
through the straits, as shown in Figure 4. Figure 5 shows the TS2 scenario as modeled in 
NABEM. 



Blue Convoy Route '•""V" 


Red Attack-East 


Red Attack-South 


Figure 4. TS2 General Laydown 
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Figure 5. TS2 NABEM Model Laydown 


The red force attacks the blue convoy from two different directions. First, a 
single Red ship attacks the convoy from the rear, hoping to draw the two Blue 
combatants away from the convoy. The second attack of four Red ships comes from the 
east and aims at the lead ship of the convoy, as shown in Figure 4. 

A. RED FORCE 

The red force is chosen to be representative of the likely threat for this tactical 
situation. Each red force platform, designated as Red PTG, was modeled as a small fast 
attack craft with operating characteristics to Peoples Republic of China Type 083 Fast 
Attack craft with a maximum speed of 35 knots, 20 nautical mile range navigational 
radar, and an ESM detection system. 

Each Red PTG carries six short-range missiles. Each missile has a maximum 
operating range of approximately 6.0 nautical miles and similar operating characteristics 
to the Hellfire missile. 
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B. BLUE FORCE 


Blue forces consist of three small surface combatants, designated as Blue SSC. 
Each Blue SSC platform is modeled as a corvette size combatant similar to the Swedish 
Goteborg Class Corvettes. Blue SSC forms the basis for the design of experiments in 
which max speed, accelerations and turning diameter are varied. 

Each Blue SSC has a medium caliber gun (similar to the Mk 57 Naval Gun), 24 
short range missiles, a point defense gun (similar to the Mk 15 CIWS), an ESM detection 
and decoy system, and a 200 nautical mile surface search radar (similar to a SPY-IF 
radar). 

Blue force convoy ships are modeled as typical coastal commercial tankers and 
were designated as Blue Tankers. 

C. METRICS FOR TACTICAL SITUATION 2 

The measures of performance for TS2 reflect measurable mission metrics that are 
influenced by the three ship characteristics being investigated, as shown in Table 6. The 
metrics are designed to evaluate the effectiveness of the Blue Forces against the threat in 
each particular scenario. 



Units 

Measure of Effectiveness 

M2.1 

Number 

Of Blue SSCs damaged by enemy attacks 

M2.2 

Number 

Of Blue SSCs sunk by enemy attacks 

M2.3 

Number 

Of Blue Tankers sunk by enemy attacks 

M2.4 

Number 

Of attacking Red PTG sunk 

Table 6. 1 

Vleasures of Effectiveness for TS2 


D. ANALYSIS FOR TACTICAL SITUATION 2 

The DoE is run in NAB EM according to the DoE matrix in Table 3. There are 15 
design points; each design point was simulated 5000 times, each with the initial 
simulation parameters randomly chosen. The results from the simulations are listed in 
Table 7. A response surface model is generated from the results in Table 7. Each 
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response is checked for RSquare values, Adjusted RSquared values, F statistics, and 
Model Fit Error to ensure the accuracy of the model fit. If any of the checks fail, it is an 
indication that the basic second-order model is not appropriate for the response. In these 
cases two options were available to provide a better fit, adding higher terms or adding 
more design points. For all the MoEs at least one of the checks failed. Higher order 
terms were tried for all the design points. Adding [Turn Diaemeter x Acceleration] to 
M2.1 and [Max Speed 2 x Acceleration] to M2.3 makes it possible to pass three of the four 
checks, but all the response surface model still fail one check, Model Fit Error. 
Generating more design points was considered, but due to time and resources was not 
possible. For this thesis the response surface models from the design points in Table 7 
are the best fit possible. For a more thorough examination the TS2’s model fit, see 
Appendix B. 

E. RESULTS 



Max Spd 

Accel 

Turn Dia 

Blue SSCs 
Damaged 

SSCs Sunk 

Tankers Sunk 

Red PTG Sunk 

(kt) 

(kt/sec) 

(«) 

Mean 

% of Total 

Mean 

% of Total 

Mean 

% of Total 

Mean 

% of Total 

1 

25 

0.5 

1000 

0.397 

(13.3) 

0.181 

(6.0) 

0.338 

(3.4) 

2.374 

(59.4) 

2 

45 

0.5 

1000 

0.432 

(14.4) 

0.228 

(7.6) 

0.205 

(2.1) 

3.449 

(86.2) 

3 

45 

0.3 

1750 

0.461 

(15.4 

0.221 

(7.4) 

0.203 

(2.0) 

3.431 

(85.8) 

4 

45 

0.1 

1000 

0.351 

(11.7) 

0.190 

(6.3) 

0.205 

(2.1) 

3.495 

(87.4) 

5 

25 

0.5 

2500 

0.582 

(19.4) 

0.238 

(7.9) 

0.337 

(3.4) 

2.376 

(59.4) 

6 

45 

0.1 

2500 

0.384 

(12.8) 

0.177 

(5.9) 

0.206 

(2.1) 

3.480 

(87.0) 

7 

35 

0.1 

1750 

0.344 

(11.5) 

0.165 

(5.5) 

0.230 

(2.3) 

2.937 

(73.4) 

8 

25 

0.3 

1750 

0.545 

(18.1) 

0.219 

(7.3) 

0.338 

(3.4) 

2.379 

(59.5) 

9 

25 

0.1 

1000 

0.402 

(13.4) 

0.184 

(6.1) 

0.345 

(3.5) 

2.382 

(59.5) 

10 

35 

0.3 

1750 

0.458 

(15.3) 

0.201 

(6.7) 

0.233 

(2.3) 

2.886 

(72.2) 

11 

35 

0.5 

1750 

0.577 

(19.2) 

0.208 

(6.9) 

0.342 

(3.4) 

2.347 

(58.7) 

12 

45 

0.5 

2500 

0.564 

(18.8) 

0.263 

(8.8) 

0.214 

(2.1) 

3.383 

(84.6) 

13 

35 

0.3 

1000 

0.408 

(13.6) 

0.177 

(5.9) 

0.245 

(2.4) 

2.912 

(72.8) 

14 

35 

0.3 

2500 

0.483 

(16.1) 

0.216 

(7.2) 

0.223 

(2.2) 

2.845 

(71.1) 

15 

25 

0.1 

2500 

0.520 

(17.3) 

0.213 

(7.1) 

0.356 

(3.6) 

2.330 

(58.2) 


Table 7. Final Results for TS2 


From the response surface model for each MoE a prediction profiler plot is 
generated which shows the interrelationships between all the parameters, Figure 6. The 
prediction profiler serves as the principal tool for evaluating the individual responses. 
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The following sections discuss the key findings for each of the ship characteristics 
investigated. 
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Figure 6. Prediction Profiler Results for TS2 

1. Max Speed 

Referring to the prediction profiler in Figure 6, maximum speed has an overall 
positive effect. As maximum speed increases the number of Blue Tankers sunk (M2.3) 
decrease and the number of PTGs sunk (M2.4) increase. This makes sense, as the 
number of PTGs goes down, so should the number of Tankers sunk because the Tankers 
are exposed to fewer PTGs and thus lowers the probability of the tankers getting sunk. 

However, as maximum speed increases, the number of Blue SSCs damaged or 
sunk (M2.1 and M2.2) increase. The explanation for this is, as speed increases Blue 
SSCs are exposed to more potential combat with Red PTGs, increasing the number of 
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Red PTGs sunk, but which also increases Blue SSCs exposure to Red weapons, 
increasing the probability of Blue SSCs getting damaged or sunk. 

While maximum speed has an effect on the outcome of the tactical situation, the 
effect is insignificant. 

2. Acceleration 

Referring to the prediction profiler in Figure 6, acceleration has an overall 
negative effect on Blue forces. As acceleration increases the number of Blue SSCs and 
Tankers sunk or damaged increase. But the number of Red PTGs sunk trends to 
decrease. This appeared to be a contradiction, but as more PTGs survive, more Blue 
ships are exposed to enemy fire, thus have a higher probability of getting damaged or 
sunk. Therefore, increasing the ship’s acceleration would increase the number of PTGs 
sunk, but in this instance the change is so small that it can be ignored. 

3. Turn Diameter 

Referring to the prediction profiler in Figure 6, turning diameter has a positive 
effect on the number Blue SSCs damaged or sunk and the number of Blue Tankers sunk 
(M2.1, M2.2, and M2.3). Also, as turning diameter gets tighter the number of Red PTGs 
sunk goes up initially, the plot for turning diameter for M2.4 as a concave structure 
indicating for a turning diameter of approximately 1750 ft the numbers of PTGs sunk are 
at a minimum. The model suggests that turning diameter was beneficial to the outcome 
of Blue forces, but these benefits are small. The small changes in the number of Red 
PTGs sunk are not significant. 

F. CONCLUSIONS 

The final response surface models developed for tactical situation 2 produced tangible 
models that link ship characteristic to mission effectiveness; however the effects are 
small and insignificant. NABEM is primarily an engineering simulation model, 
concerned more with the detailed mathematical representation of individual systems or 
components. NABEM provides a detailed representation of sensor and weapon systems, 
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not platform characteristics. For this reason sensor and weapons effects dominated the 
results. Maximum speed, acceleration and turning diameter appear as secondary effects 
producing little or no impact to the overall outcome of the tactical situation. 
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V. 


CONCLUSIONS 


Response surfaces generated from the NABEM simulations produce insignificant 
results. Maximum speed, acceleration and turning diameter produce only secondary 
effects, while sensor and weapons effects dominate the results. NABEM is primarily an 
engineering simulation model, concerned more with the detailed mathematical 
representation of individual systems or components. It provides a detailed representation 
of sensor and weapon systems, not platform characteristics. For this reason sensor and 
weapons effects dominates the results, while maximum speed, acceleration and turning 
diameter appeared as secondary effects producing little or no effect. 

Implementation of the methodology allows a designer to assess and trade-off 
impacts of various ship characteristics based on mission effectiveness. The methodology 
provides a framework where feasible and economically viable alternatives can be 
identified with accuracy along with their effects on mission effectiveness. While the 
methodology is capable of supporting the JCIDS process, NABEM is not an effective 
simulation tool to use in the process. 

Maximum speed, acceleration and turning diameter are typically tactical decisions 
made by the ship operator based on the current tactical situation, the “human in the loop”. 
If the platform sensors and weapons generate the tactical decision environment, then ship 
platform performance—such as maximum speed, acceleration, turning diameter—may 
become the dominating factors, which makes it possible to quantitatively assess their 
effects on mission effectiveness. Modeling human response to changing tactical 
situations, in relation to platform performance, is a daunting task. Such a model would 
greatly enhance future ability to assess ship platform characteristics on mission 
effectiveness and is suggested as a future research project. 
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APPENDIX A RESPONSE SURFACE MODEL FOR 
TACTICAL SITUATION 1 


The following sections provide the general procedures to develop the response 
surface model fits for each measure of effectiveness for tactical situation 1. All analysis 
work is preformed using SAS’s statistical modeling tool JMP (Version 5.1). 

A. FIT FOR Ml.l 

The results for the NAB EM tactical situation 1 simulation for Ml.2 MoE arre 
feed into JMP. The fit model option is used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.988 and 0.967 respectively, Figure A1 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
Two, the Analysis of Variance the F statistic is 0.0003, which is well below the 0.05 for a 
desired 95% confidence level. The basic model statistics look good. 

Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure Al, all the data points are falling within the 95% confidence lines (the 
dashed red line), again indicating a good fit to the data. Two, the Residual by Predicted 
Plot, Figure A2, shows a fairly evenly shattering of the points. Some areas in the plot 
have holes, but that can be attributed to the small data set being analyzed, otherwise 
everything indicates a good model fit. 

All indications show that the model listed in the Parameter Estimates in Figure Al 
is a good model fit to the data. One additional test is needed to verify the accuracy of the 
model, a model fit error analysis. 

The model fit error analysis checks how well the model fits the data points from 
the design of experiments. The model fit error, measured as the model percent error, is 
determined for each data point and the resulting distribution is evaluated against two 
error distribution criteria; a mean of approximately 0.0 and a standard deviation of less 
than 1.0 are desired. The model percent error is computed for each using the following 
equation: 


29 



{[(Predicted Value) - (Ml.l Actual)] / (Ml.l Actual)} x 100 


Figure A3 shows the resulting distribution for the model percent error. The 
distribution does not look like a normal distribution, which is what would be expected. 
In addition, the mean is 0.017, which is close to 0.0, but the standard deviation is 1.39, 
which is higher than desired. Based on the model fit error analysis, the model for Ml.l 
does not perform as well as would be expected from the initial analysis of the fit. This is 
due most likely to the small data set used for the analysis. An additional set of data 
points should be added to the analysis, to help better define the model. However, due to 
time and limited resources, that is not possible, and the fit will have to suffice with the 
caveat that the final analysis can only provide general trends, not qualitative answers. 
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Actual by Predicted Plot 



Ml .1 Predcted R=0.0003 RSq=0.99 
RMSE=0.0235 


Summary of Fit 


RSquare 

0.988398 

RSquare Adj 

0.967514 

Root Mean Square Error 

0.023519 

Mean of Ffesponse 

1.022707 

Observatbns (or SumWgts) 

15 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

9 

0.23562197 

0.026180 

47.3289 

Error 

5 

0.00276578 

0.000553 

FVob > F 

C. Total 

14 

0.23838775 


0.0003 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Profc»|t| 

Intercept 

2.2018056 

0.18119 

12.15 

<.0001 

Max Spd 

-0.100403 

0.010549 

-9.52 

0.0002 

Accel 

0.9544208 

0.283497 

3.37 

0.0200 

Turn Dia 

0.0001225 

0.000101 

1.21 

0.2797 

Max Spd*Max Spd 

0.0016942 

0.000147 

11.55 

<.0001 

Max Spd*Accel 

-0.013388 

0.004158 

-3.22 

0.0235 

AcceFAccel 

-0.780833 

0.366671 

-2.13 

0.0865 

Max SpdTurn Da 

-0.000001 

0.000001 

-1.27 

0.2614 

AcceFTurn Dia 

0.0000692 

0.000055 

1.25 

0.2674 

Turn Dia*Turn Dia 

-2.113e-8 

2.607e-8 

-0.81 

0.4546 


Figure A1. Summary of Model Fit For Ml. 1 
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Effect Tests 


Source 

Nparm 

DF 

Sum of Squares 

F F5atio 

Prob > F 

Max Spd 

1 

1 

0.05010935 

90.5882 

0.0002 

Accel 

1 

1 

0.00626944 

11.3340 

0.0200 

Turn Dia 

1 

1 

0.00081246 

1.4688 

0.2797 

Max SpcflVkx Spd 

1 

1 

0.07380516 

133.4257 

<.0001 

Max Spd*Accel 

1 

1 

0.00573521 

10.3682 

0.0235 

AccePAccel 

1 

1 

0.00250848 

4.5349 

0.0865 

Max Spd*Turn Da 

1 

1 

0.00088621 

1.6021 

0.2614 

Accel*Turn Dia 

1 

1 

0.00086112 

1.5568 

0.2674 

Turn Dia*Turn Dia 

1 

1 

0.00036312 

0.6565 

0.4546 


Residual by Predicted Plot 


0 . 02 - 1 — 

0 . 01 - 

15 

I 0 . 00 - . 

CD 

DC 
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Figure A2. Summary of Model Fit For M1.1 
(continued) 


Figure A3. Model Fit Error For 
Ml.l 


B. FIT FOR M1.2 

The results for the NAB EM tactical situation 1 simulation for the Ml.2 MoE, are 
feed into JMP. The fit model option is used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.931 and 0.808 respectively, Figure A4 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
Two, the Analysis of Variance the F statistic is 0.0191, which is well below the 0.05 for a 
desired 95% confidence level. The basic model statistics look good. 
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Second, a review of two key plots was preformed. One, the Actual by Predicted 
Plot, Figure A4, all the data points, with the exception of two points, fall within the 95% 
confidence lines (the dashed red line), indicating a good fit to the data. Two, the 
Residual by Predicted Plot, Figure A5, shows a fairly evenly shattering of the points. 
Some areas in the plot have holes, but that can be attributed to the small data set being 
analyzed, otherwise everything indicates a good model fit. 

All indications show that the model listed in the Parameter Estimates in Figure A4 
is a good model fit to the data. 

For the model fit error analysis, Figure A6 shows the resulting distribution for the 
model percent error. The distribution does not look like a normal distribution, which is 
what would be expected. In addition, the mean is 0.061, which is close to 0.0, but the 
standard deviation is 2.28, which was much higher than desired. Based on the model fit 
error analysis, the model for Ml.2 does not perform as well as would be expected from 
the initial analysis of the fit. This is due most likely to the small data set used for the 
analysis. An additional set of data points should be added to the analysis, to help better 
define the model. However, due to time and limited resources, that is not possible, and 
the fit will have to suffice with the caveat that the final analysis can only provide general 
trends, not qualitative answers. 
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Actual by Predicted Plot 



RSquare 

0.93161 

RSquare Adj 

0.808507 

Root Mean Square Error 

0.015241 

Mean of Ffesponse 

0.360433 

Observatbns (or SumWgts) 

15 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

9 

0.01582198 

0.001758 

7.5677 

Error 

5 

0.00116151 

0.000232 

FVob > F 

C. Total 

14 

0.01698349 


0.0191 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

0.1516173 

0.117419 

1.29 

0.2531 

Max Spd 

0.0098716 

0.006836 

1.44 

0.2083 

Accel 

0.1288625 

0.183718 

0.70 

0.5143 

Turn Dia 

-0.000046 

0.000065 

-0.71 

0.5112 

Max Spd*Max Spd 

-0.000073 

0.000095 

-0.77 

0.4778 

Max Spd*Accel 

0.0003125 

0.002694 

0.12 

0.9122 

AcceFAccel 

-0.195972 

0.237618 

-0.82 

0.4471 

Max SpdTurn Da 

-5.767e-7 

7.185e-7 

-0.80 

0.4586 

AcceFTurn Dia 

-0.000015 

0.000036 

-0.41 

0.7000 

Turn Dia*Turn Dia 

1.8153e-8 

1.69e-8 

1.07 

0.3318 


Figure A4. Summary of Model Fit For Ml .2 
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Error Ml.2 


Effect Tests 


Source 

Nparm 

DF 

Sum of Squares 
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1 

1 

0.00000313 

0.0135 

0.9122 

Accel*Accel 

1 

1 

0.00015801 

0.6802 

0.4471 

Max Spd*Turn Da 

1 

1 

0.00014964 

0.6442 

0.4586 

AccerTurn Dia 

1 

1 

0.00003872 

0.1667 

0.7000 

Turn Dia*Turn Dia 
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0.00026811 

1.1542 

0.3318 
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Figure A5. Summary of Model Fit For Ml.2 
(continued) 



Quantiles 


Quantiles 

100.0% maximum 6.310 

99.5% 6.310 

97.5% 6.310 

90.0% 4.029 

75.0% quartile 1.190 

50.0% median -0.421 

25.0% quartile -0.969 

10.0% -3.546 

2.5% -5.371 

0.5% -5.371 

0.0% minimum -5.371 

Moments 


Mean 

0.0616479 

Std Dev 

2.5828597 

Std Err Mean 

0.6668915 

upper 95% Mean 

1.4919879 

lower 95% Mean 

-1.368692 

N 

15 


Figure A6. Model Fit Error 
For Ml.2 


C. FIT FOR M1.3 

The results for the NAB EM tactical situation 1 simulation for the Ml.3 MoE, are 
feed into JMP. The fit model option is used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 

RSquared and RSquared Adjusted values are 0.943 and 0.840 respectively, Figure A7 

Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 

Two, the Analysis of Variance the F statistic is 0.0125, which is well below the 0.05 for a 

desired 95% confidence level. The basic model statistics look good. 
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Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure A7, all the data points, with the exception of one point, fall within the 95% 
confidence lines (the dashed red line), indicating a good fit to the data. Two, the 
Residual by Predicted Plot, Figure A8, shows a fairly evenly shattering of the points. 
Some areas in the plot have holes, but that can be attributed to the small data set being 
analyzed, otherwise everything indicates a good model fit. 

All indications show that the model listed in the Parameter Estimates in Figure A7 
is a good model fit to the data. 

For the model fit error analysis, Figure A9 shows the resulting distribution for the 
model percent error. The distribution does not look like a normal distribution, which is 
what would be expected. In addition, the mean is 0.054, which is close to 0.0, but the 
standard deviation is 2.38, which was much higher than desired. Based on the model fit 
error analysis, the model for Ml.3 does not perform as well as would be expected from 
the initial analysis of the fit. This is due most likely to the small data set used for the 
analysis. An additional set of data points should be added to the analysis, to help better 
define the model. However, due to time and limited resources, that is not possible, and 
the fit will have to suffice with the caveat that the final analysis can only provide general 
trends, not qualitative answers. 
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Actual by Predicted Plot 
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Analysis of Variance 


Source 

DF 

Sum of Squares 
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9 
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9.1903 

Error 

5 
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14 
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0.0125 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Profc»|t| 

Intercept 

1.1899542 

0.374395 

3.18 

0.0246 

Max Spd 

-0.000472 

0.021797 

-0.02 

0.9836 

Accel 

0.5637875 

0.585794 

0.96 

0.3800 

Turn Dia 

-0.000239 

0.000209 

-1.15 

0.3040 

Max Spd*Max Spd 

0.0001762 

0.000303 

0.58 

0.5863 

Max Spd*Accel 

-0.000963 

0.008591 

-0.11 

0.9152 

AcceFAccel 

-1.132083 

0.757656 

-1.49 

0.1954 

Max SpdTurn Da 
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0.000002 

0.49 

0.6447 
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0.21 
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Turn Dia*Turn Dia 

5.2919e-8 

5.388e-8 

0.98 
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Figure A7. Summary of Model Fit For Ml .3 
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Effect Tests 
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Sum of Squares 
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1 

1 
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1 

1 
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0.2404 
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Figure A8. Summary of Model Fit For M1.3 
(continued) 


Figure A9. Model Fit Error For 
Ml.3 
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APPENDIX B RESPONSE SURFACE MODEL FOR 
TACTICAL SITUATION 2 


The following sections provide the general procedures to develop the response 
surface model fits for each measure of effectiveness for tactical situation 2. All analysis 
work is preformed using SAS’s statistical modeling tool JMP (Version 5.1). 

A. FIT FOR M2.1 

The results for the NABEM tactical situation 1 simulation are feed into JMP. The 
fit model option is used to do an initial fit of the data using the surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.879 and 0.662 respectively, Figure B1 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
Two, the Analysis of Variance the F statistic is 0.0682, which is above the 0.05 for a 
desired 95% confidence level. The basic model statistics do not look great and deserve 
some more attention. 

Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure Bl, all the data points are falling within the 95% confidence lines (the 
dashed red line), indicating a good fit to the data. Two, the Residual by Predicted Plot, 
Figure B2, shows a fairly evenly shattering of the points. Some areas in the plot have 
holes, but that can be attributed to the small data set being analyzed, otherwise the plots 
indicate good model fit. 

To fix the fit statistics, higher order terms (HOT) are added to the model. The 
two terms that have the most influence from the initial fit are (Turn Dia) and (Accel) 
(refer to the Pareto Plot in Figure B3). Running combinations of higher order terms using 
Turn Dia and Accel, the best results are achieved by adding the term [Turn Dia * Turn 
Dia * Accel]. The new RSquared and RSquared Adjusted values are 0.981 and 0.933 
respectively, and the Analysis of Variance the F statistic is 0.0053, Figure B4. 

The new Actual by Predicted Plot (Figure B4) and Residual by Predicted Plot 
(Figure B5) show the same similar characteristics as the initial fit 
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All indications show that the model listed in the Parameter Estimates in Figure B4 
is a good model fit to the data. One additional test is needed to verify the accuracy of the 
model, a model fit error analysis. 

The model fit error analysis checks how well the model fits the data points from 
the design of experiments. The model fit error, measured as the model percent error, is 
determined for each data point and the resulting distribution is evaluated against two 
error distribution criteria; a mean of approximately 0.0 and a standard deviation of less 
than 1.0 are desired. The model percent error is computed for each using the following 
equation: 

{[(Predicted Value) - (Ml.l Actual)] / (Ml.l Actual)} x 100 

Figure 6 shows the resulting distribution for the model percent error. The 
distribution does not look like a normal distribution, which is what would be expected. 
In addition, the mean is 0.034, which is close to 0.0, but the standard deviation is 2.46, 
which is higher than desired. Based on the model fit error analysis, the model for M2.1 
does not perform as well as would be expected from the initial analysis of the fit. This is 
due most likely to the small data set used for the analysis. An additional set of data 
points should be added to the analysis, to help better define the model. However, due to 
time and limited resources, that is not possible, and the fit will have to suffice with the 
caveat that the final analysis can only provide general trends, not qualitative answers. 
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Actual by Predicted Plot 



Summary of Fit 


RSquare 

0.879522 

RSquare Adj 

0.662663 

Root Mean Square Error 

0.047242 

Mean of Response 

0.46052 

Observatbns (or SumWgts) 

15 

Analysis of Variance 


Source 
Model 
Error 
C. Total 


DF Sum of Squares 
9 0.08146320 

5 0.01115888 

14 0.09262208 


Mean Square F Ratio 

0.009051 4.0557 

0.002232 Prob > F 
0.0687 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

0.562202 

0.363946 

1.54 

0.1831 

Max Spd 

-0.021751 

0.021189 

-1.03 

0.3517 

Accel 

-0.195535 

0.569444 

-0.34 

0.7453 

Turn Dia 

0.0002962 

0.000203 

1.46 

0.2042 

Max Spd*Max Spd 

0.0002767 

0.000295 

0.94 

0.3907 

Max Spd*Accel 

0.0128187 

0.008351 

1.53 

0.1854 

Accel*Accel 

-0.364444 

0.736509 

-0.49 

0.6417 

Max SpdTurn Da 

-0.000002 

0.000002 

-1.03 

0.3513 

AccePTum Dia 

0.0001381 

0.000111 

1.24 

0.2700 

Turn Dia*Turn Dia 

-5.294e-8 

5.237e-8 

-1.01 

0.3585 


Effect Tests 


Source Nparm 

DF Sum of Squares 

F Ratio 

Prob > F 

Max Spd 1 

1 0.00235164 

1.0537 

0.3517 

Accel 1 

1 0.00026315 

0.1179 

0.7453 

Turn Dia 1 

1 0.00475420 

2.1302 

0.2042 

Max Spd*IVbx Spd 1 

1 0.00196908 

0.8823 

0.3907 

Max Spd*Accel 1 

1 0.00525825 

2.3561 

0.1854 

Accel*Accel 1 

1 0.00054646 

0.2449 

0.6417 

Max Spd*Turn Da 1 

1 0.00235641 

1.0558 

0.3513 

Accel*Turn Dia 1 

1 0.00343206 

1.5378 

0.2700 

Turn Dia*Turn Dia 1 

1 0.00228013 

1.0217 

0.3585 
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Figure B1. Summary of Initial Model Fit For M2.1 


Figure B2. Summary of Initial Model Fit For M2.1 (continued) 


Pareto Plot of Transformed Estimates 


Term 

Accel 

Turn Dia 

Max Spd 

Max Spd*Accel 

Accel*Turn Dia 

Max Spd*Turn Dia 

Turn Dia*Turn Dia 

Accel*Accel 

Max Spd*Max Spd 


Orthog Estimate 
0.0450869 
0.0442541 
-0.0207308 
0.0187230 
0.0151263 
-0.0125337 
-0.0123292 
-0.0099742 
0.0046810 



Figure B3. Pareto Plot For M2.1 Initial Fit 
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Actual by Predicted Plot 



RSquare 

0.980716 

RSquare Adj 

0.932507 

Root Mean Square Error 

0.021131 

Mean of Response 

0.46052 

Observations (or Sum Wgts) 

15 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

10 

0.09083598 

0.009084 

20.3428 

Error 

4 

0.00178610 

0.000447 

Prob > F 

C. Total 

14 

0.09262208 


0.0053 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

1.0954133 

0.200116 

5.47 

0.0054 

Max Spd 

-0.021751 

0.009478 

-2.29 

0.0834 

Accel 

-1.972906 

0.464087 

-4.25 

0.0131 

Turn Dia 

-0.000418 

0.00018 

-2.32 

0.0814 

Max Spd*Max Spd 

0.0002767 

0.000132 

2.10 

0.1037 

Max Spd*Accel 

0.0128187 

0.003735 

3.43 

0.0265 

Accel* Accel 

-0.364444 

0.32944 

-1.11 

0.3307 

Max Spd*Turn Dia 

-0.000002 

9.961 e-7 

-2.30 

0.0832 

Accel*Turn Dia 

0.0025193 

0.000522 

4.83 

0.0085 

Turn Dia*Turn Dia 

1.5116e-7 

5.033e-8 

3.00 

0.0398 

Turn Dia*Turn Dia* Ac cel 

-6.803e-7 

1.485e-7 

-4.58 

0.0102 


Figure B4. Summary of Model Fit For M2.1 HOT 
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Effect Tests 


Source 

Nparm 

DF 

Sum of Squares 

F Ratio 

Prob > F 

Max Spd 

1 

1 

0.00235164 

5.2665 

0.0834 

Accel 

1 

1 

0.00806974 

18.0723 

0.0131 

Turn Dia 

1 

1 

0.00239826 

5.3709 

0.0814 

Max Spd*Max Spd 

1 

1 

0.00196908 

4.4098 

0.1037 

Max Spd*Accel 

1 

1 

0.00525825 

11.7759 

0.0265 

Accel *Accel 

1 

1 

0.00054646 

1.2238 

0.3307 

Max Spd*Turn Dia 

1 

1 

0.00235641 

5.2772 

0.0832 

AcceFTurn Dia 

1 

1 

0.01039588 

23.2817 

0.0085 

Turn Dia*Turn Dia 

1 

1 

0.00402745 

9.0195 

0.0398 

Turn Dia*Turn Dia*Accel 

1 

1 

0.00937278 

20.9905 

0.0102 
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97.5% 

90.0% 

75.0% quartile 
50.0% median 
25.0% quartile 
10.0% 

2.5% 

0.5% 

0.0% minimum 

Moments 


3.314 

3.314 

3.314 

3.099 

2.790 

-0.587 

-0.984 

-4.176 

-4.961 

-4.961 

-4.961 



Mean 
Std Dev 
Std Err Mean 
upper 95% Mean 
lower 95% Mean 
N 


0.0339701 

2.4615782 

0.6355768 

1.3971467 

-1.329206 

15 


Figure B5. Summary of Model Fit For M2.1 HOT (continued) 


Figure B6. Model Fit Error For M2.1 HOT 


B. FIT FOR M2.2 

The results for the NABEM tactical situation 2 simulations for the M2.2 MoE, are 
feed into JMP. The fit model option is used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.984 and 0.953 respectively, Figure B4 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
Two, the Analysis of Variance the F statistic is 0.0006, which is well below the 0.05 for a 
desired 95% confidence level. The basic model statistics look good. 
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Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure B4, all the data points fall within the 95% confidence lines (the dashed red 
line), indicating a good fit to the data. Two, the Residual by Predicted Plot, Figure B5, 
shows a fairly evenly shattering of the points. Some areas in the plot have holes, but that 
can be attributed to the small data set being analyzed, otherwise everything indicates a 
good model fit. 

All indications show that the model listed in the Parameter Estimates in Figure B4 
is a good model fit to the data. 

For the model fit error analysis, Figure B6 shows the resulting distribution for the 
model percent error. The distribution does not look like a normal distribution, which is 
what would be expected. In addition, the mean is 0.035, which is close to 0.0, but the 
standard deviation is 1.73, which is much higher than desired. Based on the model fit 
error analysis, the model for M2.2 does not perform as well as would be expected from 
the initial analysis of the fit. This is due most likely to the small data set used for the 
analysis. An additional set of data points should be added to the analysis, to help better 
define the model. However, due to time and limited resources, that is not possible, and 
the fit will have to suffice with the caveat that the final analysis can only provide general 
trends, not qualitative answers. 
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Actual by Predicted Plot 



RSquare 

0.983524 

RSquare Adj 

0.953868 

Root Mean Square Error 

0.005753 

Mean of Ffesponse 

0.205393 

Observatbns (or SumWgts) 

15 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

9 

0.00987774 

0.001098 

33.1639 

Error 

5 

0.00016547 

0.000033 

FVob > F 

C. Total 

14 

0.01004321 


0.0006 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Profc»|t| 

Intercept 

0.4019033 

0.044319 

9.07 

0.0003 

Max Spd 

-0.015339 
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Figure B7. Summary of Model Fit For M2.2 
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Figure B8. Summary of Model Fit For M2.2 (continued) 


Figure B9. Model Fit Error 
For M2.2 


C. FIT FOR M2.3 

The results for the NABEM tactical situation 2 simulations for the M2.3 MoE, are 
feed into JMP. The fit model option is used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.892 and 0.698 respectively, Figure B10 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
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Two, the Analysis of Variance the F statistic is 0.0539, which is above the 0.05 for a 
desired 95% confidence level. The basic model statistics do not look great and deserve 
some more attention. 

Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure BIO, all the data points, with the exception of two points, fall within the 95% 
confidence lines (the dashed red line), indicating a good fit to the data. Two, the 
Residual by Predicted Plot, Figure Bll, shows a fairly evenly shattering of the points. 
Some areas in the plot have holes, but that can be attributed to the small data set being 
analyzed, otherwise everything indicates a good model fit. 

To fix the fit statistics, higher order terms are added to the model. The two terms 
that have the most influence from the initial fit are (Turn Dia) and (Accel) (refer to the 
Pareto Plot in Figure B12). Running combinations of higher order terms using Turn Dia 
and Accel, the best results are achieved by adding the term [Turn Dia * Turn Dia * 
Accel]. The new RSquared and RSquared Adjusted values are 0.987 and 0.954 
respectively, and the Analysis of Variance the F statistic was 0.0025, Figure B13. 

The new Actual by Predicted Plot (Figure B13) and Residual by Predicted Plot 
(Figure Bll) show the same similar characteristics as for the initial fit 

All indications show that the model listed in the Parameter Estimates in Figure 
B13 was a good model fit to the data. 

For the model fit error analysis, Figure B15 shows the resulting distribution for 
the model percent error. The distribution does not look like a normal distribution, which 
is what would be expected. In addition, the mean is 0.035, which is close to 0.0, but the 
standard deviation is 1.73, which was much higher than desired. Based on the model fit 
error analysis, the model for M2.2 does not perform as well as would be expected from 
the initial analysis of the fit. This is due most likely to the small data set used for the 
analysis. An additional set of data points should be added to the analysis, to help better 
define the model. However, due to time and limited resources, that is not possible, and 
the fit will have to suffice with the caveat that the final analysis can only provide general 
trends, not qualitative answers. 
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Figure BIO. Summary of Initial Model 
Fit For M2.3 


Figure B11. Summary of Initial Model Fit For M2.3 
(continued) 
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Figure B12. Pareto Plot For M2.3Initial Fit 
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Actual by Predicted Plot 
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Figure B13. Summary of Model Fir M2.3 HOT 
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Error M2.3 HOT 
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Figure B14. Summary of Model Fit For M2.3 HOT 
(continued) 
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Figure B15. Model Fit 
Error For M2.3 HOT 


D. FIT FOR M2.4 

The results for the NABEM tactical situation 2 simulations for the M2.4 MoE, are 
feed into JMP. The fit model option was used to do an initial fit of the data using the 
surface response option. 

First is to determine if the model statistics indicate a good fit to the data. One, the 
RSquared and RSquared Adjusted values are 0.956 and 0.877 respectively, Figure B16 
Generally anything higher than 0.80 would be acceptable and indicates a good model fit. 
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Two, the Analysis of Variance the F statistic is 0.0067, which is well below the 0.05 for a 
desired 95% confidence level. The basic model statistics look good. 

Second, a review of two key plots is preformed. One, the Actual by Predicted 
Plot, Figure B16, all the data points, with the exception of two points, fall within the 95% 
confidence lines (the dashed red line), indicating a good fit to the data. Two, the 
Residual by Predicted Plot, Figure B17, shows a fairly evenly shattering of the points. 
Some areas in the plot have holes, but that can be attributed to the small data set being 
analyzed, otherwise everything indicates a good model fit. 

All indications show that the model listed in the Parameter Estimates in Figure 
B10 is a good model fit to the data. 

For the model fit error analysis, Figure B18 shows the resulting distribution for 
the model percent error. The distribution does not look like a normal distribution, which 
is what would be expected. In addition, the mean is 0.141, which is close to 0.0, but the 
standard deviation is 3.85, which was much higher than desired. Based on the model fit 
error analysis, the model for M2.2 does not perform as well as would be expected from 
the initial analysis of the fit. This is due most likely to the small data set used for the 
analysis. An additional set of data points should be added to the analysis, to help better 
define the model. However, due to time and limited resources, that is not possible, and 
the fit will have to suffice with the caveat that the final analysis can only provide general 
trends, not qualitative answers. 
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Figure B16. Summary of Model Fit For M2.4 


52 






























Effect Tests 


Source 

Nparm 

DF 

Sum of Squares 

F FJatio 

Prob > F 

Max Spd 

1 

1 

0.00593924 

0.2105 

0.6656 

Accel 

1 

1 

0.02786401 

0.9877 

0.3659 

Turn Dia 

1 

1 

0.02411475 

0.8548 

0.3976 

Max Spd*IVkx Spd 

1 

1 

0.04523176 

1.6033 

0.2612 

Max Spd*Accel 

1 

1 

0.00412686 

0.1463 

0.7178 

Accel*Accel 

1 

1 

0.04404224 

1.5612 

0.2668 

Max Spd*Turn Da 

1 

1 

0.00011781 

0.0042 

0.9510 

Accel*Turn Dia 

1 

1 

0.00000010 

0.0000 

0.9986 

Turn Dia*Turn Dia 

1 

1 

0.02879876 

1.0208 

0.3587 


Residual by Predicted Plot 


0.3 


0 . 2 - 
I 0.1 



- 0 . 2 - 


j i-1-1-1-1-1- 

2.25 2.50 2.75 3.00 3.25 3.50 3.75 

M2.4 Predicted 


Error M2.4 



Quantiles 


100.0% 

maximum 

10.58 

99.5% 


10.58 

97.5% 


10.58 

90.0% 


5.88 

75.0% 

quartile 

1.83 

50.0% 

median 

0.39 

25.0% 

quartile 

-2.18 

10.0% 


-4.66 

2.5% 


-6.90 

0.5% 


-6.90 

0.0% 

minimum 

-6.90 


Moments 


Figure B17. Summary of Model Fit For M2.4 (continued) 
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Figure B18. Model Fit 
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